Information-based inference in sloppy and singular models

The failure of the information-based Akaike Information Criterion (AIC) in the context of singular models can be rectified by the definition of a Frequentist Information Criterion (FIC). FIC applies a frequentist approximation to the computation of the model complexity, which can be estimated analytically in many contexts. Like AIC, FIC can be understood as an unbiased estimator of the model predictive performance and is therefore identical to AIC for regular models in the large-observationnumber limit (N →∞) . In the presence of unidentifiable parameters, the complexity exhibits a more general, non-AIC-like scaling ( N). For instance, both BIC-like (∝ logN ) and Hannan-Quinn-like (∝ log logN ) scaling with observation number N are observed. Unlike the Bayesian model selection approach, FIC is free from ad hoc prior probability distributions and appears to be widely applicable to model selection problems. Finally we demonstrate that FIC (information-based inference) is equivalent to frequentist inference for an important class of models.